Regression testing on derecho is underway.

Perfectly fine @glemieux and up to you. Since this is a tag on master there is no need to make it b4b. So do what you need to do.

Regression testing aux_clm on izumi and derecho is underway.

Regression testing on derecho is complete. I discovered NGEET/fates#1324 during testing and have updated the expected failure test list accordingly. Also I ran into #2930 again, so I reran the suite omitting the --testid-base option for the system tests.

Expected fates tests are b4b with the exception of tests that use Clm60Fates based compsets, which is as expected due to using ZengWang2007 instead of Meier2022 per this PR. An additional three tests are showing DIFFs for the FATES_GDD variable only. @rgknox did you see this during testing of NGEET/fates#1306?

FAIL ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdTwoStream BASELINE ctsm5.3.021: DIFF
FAIL ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdFixedBiogeo BASELINE ctsm5.3.021: DIFF
FAIL SMS_D_Lm6_P256x1.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesCold BASELINE ctsm5.3.021: DIFF

All expected clm tests are b4b except for SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop. This is failing during RUN with the following in the Teststatus:

153  ---------------------------------------------------
154 2025-02-03 00:19:44: ERROR: Model did not complete - see /glade/u/home/glemieux/scratch/ctsm-tests/tests_0202-192217de/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.0202-192217de_nvh/    run/med.log.7645521.desched1.250203-001918
155 
156  ---------------------------------------------------

Looking at the med.log I don't see anything in particular jumping out at me, but I don't review this log often.

I haven't been able to check izumi results due to what I'm assuming are continued issues seen from this past week.

Results : /glade/u/home/glemieux/scratch/ctsm-tests/tests_0202-192217de

#1733 discusses this test SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop

Per discussion with @rgknox at the standup this morning, it is likely that NGEET/fates#1306 is the cause of the FATES_GDD differences seen above. To confirm, I ran one of the above tests using the fates tag associated with that PR ,fates-sci.1.80.9_api.37.0.0-ctsm5.3.018, and compared it against the previous baseline. The results are consistent and can be found below.

Results: /glade/u/home/glemieux/ctsm/tests_0203-114755de

As such, testing on derecho is complete. Just waiting on izumi to come back up to check tests there.

Per discussion with @rgknox, I've removed FATES_C13_DISC_SZPF from AllVars as this is a known low priority issue (see NGEET/fates#700). I've rerun the AllVars test and regenerated that specific baseline.

Rerunning tests on izumi as it seems some tests were mangled during last Friday's issues.

Rerunning aux_clm izumi tests again as a number of them failed due to input data still not being available yesterday evening.

Resubmitting failed aux_clm cases on izumi via --rerun-existing-failures seem to have mostly worked. Nearly all expected tests are B4B. The only DIFFs are for fates testmods and are expected due either to the z0param_method update or due to a fates update to the temperature average (i.e. the FATES_GDD diffs mentioned above).

Three tests did not manage to generate baselines due to lack of space:

(ctsm_pylib) [glemieux@izumi tests_0204-101554iz]$ ./cs.status -f | grep -i generate
    FAIL ERS_Ld600_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-cropMonthlyNoinitial GENERATE exception
    FAIL SMS_Ld5_Mmpi-serial.1x1_brazil.IHistClm60Bgc.izumi_gnu.clm-mimics GENERATE exception
    FAIL SMS_Ln9.f10_f10_mg37.I1850Clm45Bgc.izumi_gnu.clm-clm45cam4LndTuningModeZDustSoilErod GENERATE exception

Aside from that, this PR is ready to merge.

Results: /home/glemieux/ctsm/tests_0204-101554iz